Summary of Changes

Hello @Muqi1029, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical memory leak within the SGLang Engine's mixed chunking mode. Previously, when a request finished during the initial prefill stage, the req_pool_idx resource was not properly released, leading to memory exhaustion and system instability. The change introduces a specific call to free this resource, thereby preventing errors detected by the scheduler and enhancing the overall robustness of the engine.

Highlights

• Memory Leak Fix: Addresses a critical memory leak that occurred in the mixed chunking mode when a request completed during the initial prefill stage, where the req_pool_idx was not being freed.
• Resource Deallocation: Ensures proper resource management by explicitly freeing the req_pool_idx using self.req_to_token_pool.free(req.req_pool_idx) within the process_batch_result_prefill function.
• System Stability: Prevents the SGLang Engine from failing due to memory exhaustion errors detected by the scheduler, significantly improving its stability.

not work, still have memory leak

@MichelleWu351 Maybe the situation of memory leak is not the same as yours. I test it in the latest sglang, it still happens.

I can easily reproduce this bug about memory leak:

Using the latest sglang:

Server

python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 8888 --enable-mixed-chunk

Client

import requests
import json

url = "http://127.0.0.1:8888/generate"

headers = {
    "Content-Type": "application/json",
    "Authorization": "Just Keep Me"
}

data = {
    "input_ids": [1] * 40957,
}

response = requests.post(url, headers=headers, json=data)

if response.status_code == 200:
    result = response.json()
    print(json.dumps(result, ensure_ascii=False, indent=2))
else:
    print(f"Error: {response.status_code}")
    print(response.text)

Result of server:

A bad request will make sglang engine fail to check memory leak, leading to the following error, making the whole sglang engine down:

[2025-10-25 07:57:04] The server is fired up and ready to roll!
[2025-10-25 07:57:12 TP0] Input length (40957 tokens) exceeds the maximum allowed length (40954 tokens). Use a shorter input or enable --allow-auto-truncate., self.rid='4781a1d1797f4ef78da4d4e4f3949168'
[2025-10-25 07:57:12 TP0] Prefill batch [10], #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-10-25 07:57:12] [http_server] Error: Input length (40957 tokens) exceeds the maximum allowed length (40954 tokens). Use a shorter input or enable --allow-auto-truncate.
[2025-10-25 07:57:12 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/root/projects/dev_project/sglang/python/sglang/srt/managers/scheduler.py", line 2769, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/root/.envs/dev/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/projects/dev_project/sglang/python/sglang/srt/managers/scheduler.py", line 1018, in event_loop_overlap
    self.self_check_during_idle()
  File "/root/projects/dev_project/sglang/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 214, in self_check_during_idle
    self.check_memory()
  File "/root/projects/dev_project/sglang/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 149, in check_memory
    self._check_req_pool()
  File "/root/projects/dev_project/sglang/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 135, in _check_req_pool
    raise ValueError(msg)
ValueError: req_to_token_pool memory leak detected!available_size=4095, total_size=4096

Reason

A request with an input_ids length below the model configuration limit can bypass the length check in the HTTP server:

However, it fails later in the engine:

This causes the request to be marked as aborted. The aborted request is then added to the waiting_queue as if it were a normal one by setting its input_ids to [0]. As a result, its out_cache_loc and request slot are allocated but never released:

Eventually, this leads to a check_memory failure, which brings down the entire engine and reduces SGLang’s robustness to malformed or edge-case requests.

DO NOT MERGE since it may be incompatible with retracted req, I am fixing it now

Thanks for bringing this. I think this issue has been solved by #12312 and #12224.

In the previous implementation, we set abort and set the finish reason at the same time, which is not compatible with the regular finish check and the overlap scheduler.

But now, we set to_finish instead of finish_reason for those requests which are going to be aborted but still need to enter the event loop.